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SUMMARY 



An important goal of research on teaching is the generation 
of principles and procedures which result in the improvement of 
instruction. It is one thing to state a theoretical principle of 
learning or a characteristic of programs which can be shown to operate 
in effective instructional sequences j it is quite another for 
teachers, or programers, to translate these statements into 
procedures for developing more effective instructional programs. Yet 
the usefulness of teaching principles lies exactly in their power to 
improve the effectiveness of instruction. 



The design of most experiments which compare program 
characteristics and principles provide no basis for assessing if 
programers, other than the experimenter, can incorporas-e the 
characteristics into their programs with good effects. A design 
which would provide for such an assessment should show that a number 
of programers following some prescribed procedure can independently 
produce consistently superior programs. By providing the means j.or 
assessing the variance obtaining from programer differences, a body 
of procedural techniques for effective programing might readily 
evolve. 



The object of this study was to conduct a pilot experiment 
applying an experimental model which identified programers as the 
experimental units of interest, the procedures given them to follow 
(instructions) as the independent variable, and the effectiveness of 
their products as the dependent variable. The substantive question 
raised in the study concerned whether a procedure for revising an 
instructional program could be developed which would lead a sample of 
programers to produce revised programs which were reliably more 
effective than (1) the original program before revision, and (2) 
revisions produced by a sample of programers following some 
reasonable alternative procedure. 



The experiment consisted of an after-only comparison of the 
effectiveness of programs prepared by two independent samples of 
programers following two sets of procedural instructions. The two 
procedures compared may be described as a ’’subjective vs. an 
’’objective” procedure for analyzing a program as a basis for revising 
it. The relationship of academic preparation and prior teaching 
experience to the programer 's effectiveness were further questions of 

interest in the study. 



A 25-minute slide- tape program on the subject of English 
Money was developed to serve as the base program. Two statements of^ 
the program’s objectives were prepared, one of which provided general 
statements which could be used to generate a large population of test 
items, and the other of which provided a specific sample of test 
items. The base program was administered to a sample of sixth grade 
students together with the sample test items. The data obtained were 
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analyzed for difficulty and discrimination power to select the items 
which were used as the post-test of the program. 

Twenty teacher-subjects (^'s) were then recruited, each of 
whom was to prepare a 15 -minute videotaped supplement for use 
following the base program. The ^'s were randomly assigned to two 
treatment groups and each directed to appear for a day at the 
television studio to plan and produce the lesson. The conditions 
under which the lessons were prepared were standardized for all ^‘s: 
no ^ was T.nformed of the subject matter until arrival for production; 
all ^'s were limited to a maximum of six hours production time from 
start to finished tape; all work was performed in the studio from the 
materials provided by the experimenter; only technical assistance was 
provided by the television staff (directorial assistance being 
limited only to correcting gross errors in content) ; and tapes were 
limited to 15-minutes in length as an absolute maximum. 

The instructions given to Treatment Group I (subjective) 
directed the programers to study the general objectives of the base 
program as characterized by the item-generating statements, then to 
view the base program estimating from their own teaching experience 
what additional instruction was most needed to enhance the effect of 
the total lesson. Based upon this estimate, then, they were to plan 
a 15-minute supplementary lesson to be recorded on videotape which 
would subsequently be used with the base program to teach a sample of 
sixth graders. 

The instructions given to Treatment Group II (objective) 
were essentially the same except that the final test items were 
provided together with data showing the percentage of sixth graders 
who actually succeeded on each item after a trial run of the base 
program. 



Twenty videotaped supplements were thus prepared and 
subsequently randomly assigned to 20 sixth-grade classes for 
administration. Each administration involved presenting a standard 
introduction, the base program, one of the twenty videotaped 
supplements, and a standard post-test under time-limited conditions. 

The effectiveness score assigned to each program was based 
upon the mean post- test score of the class in which the program was 
shown. As each program was shown in only one randomly assigned class, 
N = 11 for Treatment Group I, N = 9 for Treatment Group II, and N = 7 
for the Base Program Only Group. 

To test the null hypothesis against the alternative that the 
magnitude of the effects produced by the different treatment groups 
occurred in a predicted order, a special application of the statistic 
S was employed. The scores obtained from all the treatment groups 
revealed that the revisions based upon an analysis of behavioral 
objectives were significantly more effective than the original version 
of the program. Moreover, the revisions based upon an objective 
analysis of test data obtained from a trial run of the original 



program were significantly more effective than the revisions based 
solely upon a subjective analysis of the objectives. 

An analysis of the scores in relation to the amount of the 
programers ' academic preparation and teaching experience failed to 
reveal any statistically significant correlation. 

It was concluded that (1) the two revision procedures 
employed in the experiment led programers to produce revised programs 
which were more effective than the original, (2) of the two 
procedures employed, the one which provided empirical test data from 
a tryout of the original program led to the production of more 
effective revisions than the procedure which provided only general 
statements of the program's objectives, (3) the experimental model 
employed in the study appears to be a viable procedure for assessing 
the relative effectiveness of alternative programing techniques. 
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INTRODUCTION 



Background of the Study 

General — An important goal of research on teaching is the 
generation of principles and procedures which result in the 
improvement of instruction. The findings of teaching research should 
provide to persons who produce teaching materials --teachers, ^^i m 
Lkers, script writers, editors, hereinafter termed «°g>^amers -- 
with viable procedures by which their products, termed programs, 
might reliably be improved. If the principles which evolve from 
teaching research cannot be demonstrated to improve tie effectiveness 
of programer performance, the usefulness of our research efforts may 
be severely limited. 

It is one thing to state a theoretical principle of 
learning, or a characteristic of programs, which can be shown to 
ooerate in effective programs; it is quite another for programers to 
transform these statements into procedures by which they can 
consistently improve their products. Yet the use u ness o sue 
statements lies exactly in their power to be translated into 
procedures which can improve the programer *s effectiveness. There is 
little evidence, however, that the principles of learning and the 
characteristics of programs being generated by our research effor 



can. 



in fact, be used with good effects by others. 



Most of the research relevant to instructional program 
variables has come out of the instructional media fields. The 
instructional media, particularly self-itistructional programs afford 
excellent tools for the systematic application of learning and 
programing principles. By standardizing stimulus sequences, the 
exp!rtaenler is provided greater control over certain classes of 
inLructional variables than is usually feasible in the classroom. 

For the most part, experiments employing these media have 
compared two or more versions of a program in which some program 
chaLcteristic has been systematically varied by the programer. ®e 
characteristic varied is usually described as an implementation of 
some learning principle. In this way, the researcher hopes to 
generate evidence supporting or refuting the validity of the 
principle (see Figure 1). 

A popular experiment with instructional programs, for 
example, compares the effects of variations in "active response. 
Postulating that students who respond actively more frequently during 
instruction learn better than those who respond actively less 
freauently, the experimenter develops two programs which presumably 
difTeToJly with rLpect to the amount of "active responding" they 
occasion. The programs are then administered to students and 
measures of their learning are obtained and compared. If the students 
are found to differ on the dependent measure, the difference is 
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attributed to the different amounts of "active response” engaged in 
by the learners (see Lumsdaine, 13, pp. 609-15). 

The conclusion to which we are led, of course, is that 
programs which occasion more active response mediate learning better 
than those which occasion less active response. However, the 
experimental model employed above allows little formal basis for 
such a generalization. Any number of alternative hypotheses might 
be offered to explain the difference between any two programs, e.g. 
the programer "felt better" writing one than the other, he wrote a 
biased program favoring the desired outcome, one program contained 
additional or different stimulus materials which produced variations 
other than "active response," and so on. In point of fact, an 
experiment which compares only two programs provides no basis upon 
which to reject the null hypothesis that one program was superior 
to the other simply by chance. 

An Experimental Model for Comparing Programing Procedures -- 
Generally, we hypothesize that implementing some instructional 
principle will have a beneficial effect upon learning, i.e. can lead 
programers to increase the effectiveness of their products. 

Therefore, we should seek to generalize the viability of the 
principle across some population of programers and their products. 

To this end our experiments should provide some formal 
basis for generalizing to this population. If some instructional 
principle is to be shown to have beneficial effects upon learning, 
it must be shown that procedures for its emplo 5 rment can be 
communicated to independent samples of programers with consistent 
good effects. It must be shown that a number of programers following 
these procedures can independently produce programs which are 
consistently superior to those produced by other programers following 
alternative procedures. Only then can the variance obtaining from 
individual programer differences be formally assessed. 

This suggests an experimental model in which programers 
are the experimental units, the effectiveness of their products 
is the dependent variable, and the procedural instructions for 
implementing the programing principle constitute the experimental 
variable of interest (see Figure 2) . (See Lumsdaine, 14 ^p. 386.) 

It was the purpose of this study to employ such a model 
in a trial experiment assessing the effectiveness of certain 
programing procedures. If it could be shown that the instructions 
given to a sample of programers could materially improve their 
products, future programing research might profitably be directed 
toward comparative studies of alternative programing procedures. 

An Application to Compare Program Revision Procedures — In 
the application of this experimental model to a specific problem, it 
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Experimental Model Employed to Compare Programing Procedures 
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was proposed to compare two procedures for revising an audio-visual 
instructional program to improve its effectiveness. 

Traditionally, developmental revision of instructional 
materials has been based upon the subjective analysis of authoritative 
persons. The revision of pre-publication drafts of instructional 
materials by subjective analysis, for example, has been common 
practice at many levels of program development activity. At the 
commercial level, for example, editors and publishers of textbooks may 
be found generally to follow this traditional procedure by employing 
“experts'* to consult with them upon the effectiveness of the materials 
at several points in their development. These consultations form the 
basis for corrective modifications of estimated weaknesses in the 
materials. At the classroom level, on the other hand, teachers will 
be found to "preview" educational films and to base their 
supplementary instruction upon subjective estimates of the film's 
ef fectiveness . 

This type of subjective procedinre has been challenged by 
advocates of more empirical analysis procedures (see Rothkopf, 23; and 
AERA, 5 ) . Given a clear statement of the effects a program 
is intended to produce, they say, the more appropriate measure of its 
effectiveness is an analysis of the program's observed effects upon 
the behavior of students. They advance the notion that the program's 
objectives ought to be stated so as to imply clearly a test of learned 
performance and that the analysis of student responses in the test 
situation is an effective basis upon which to correctively modify the 
program (see Cropper, et al, 4 ). 

While the principal issue of this study lies in the 
application of a particular experimental model for comparing the 
effects of alternative programing procedures, the selection of a 
problem in the area of analysis-for-revision may be seen to have 
important implications for a wide range of programers from the 
classroom teacher to the mass-media programer. In the creation of 
instructional materials, judgements concerning their ef fectiveness are 
made more or less continuously upon which the materials are 
subsequently modified. In this sense the task of the programer 
seeking to revise an early draft of a program is little different 
from the task of the teacher seeking to supplement some instructional 
material for her students. Each assumes that the product in hand is 
insufficient for producing all the desired effects and each seeks to 
modify the material on the basis of an estimate of the nature of that 
insufficiency. Each has in mind some set of objectives, either 
expressed or implied, and each acts to modify the material to better 
achieve these objectives. 

In view of the great emphasis self-instructional programers 
place upon the analysis of student response data as a means for 
revising programs, empirical evidence might be thought to abound 
substantiating the ef fectiveness of such procedures. In spite of the 
apparent number of programs that have been produced by such methods. 
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however, there is little experimental evidence that the final versions 
of the programs so produced were in fact more effective than the 
initial versions. Further, even granting that the final versions were 
superior, there is virtually no evidence to support the claim that ^ 
these procedures produced final versions superior to those which might 
have been produced by a more traditional and less costly procedure. 
Finally, and with particular regard to the principal issue of this 
study, there is no evidence that any specific procedure for revising a 
program can be communicated to others such that the progr^ers who 
follow it may reliably be expected to improve upon the original. 

The proposed study may be described as a comparison of two 
procedures, one "subjective" and one "objective" for improving 
programer effectiveness in revising an instructional program. 

Review of Related Research 

A review of the experimental literature dealing with 
procedures for analyzing programs as a basis for revision was 
conducted to provide guidance in designing the experimental materials 
and procedures which would be used in the study. 

Audiovisual programs have been modified on the basis of 
analysis of student response data by a number of researchers who 
report upon the effectiveness of their procedures. Most of these 
studies have been conducted to effect two kinds of outcomes upon an 
intended audience: (1) attitude change and, (2) cognitive gain. Most 
of the formal effort has been expended in the former. 

Sheffield (24) conducted extensive prior interviewing 
among an intended soldier population to determine the bases of 
resistance to attitude change. His analysis of the soldiers 
responses to the interview questions was employed in the preparation 
of an audio recording intended to modify soldier attitudes about the 
probable length of the war. 

The procedure involved, among other "rules," an attempt to 
introduce negative arguments at those points where they seemed likely 
to occur spontaneously to the opposed members of the audience. These 
points were determined empirically by pretesting a sample of the 
expected audience by means of individual interview techniques. In 
these pretests, interviewers spoke individually to the soldiers and 
actually presented the case to be advocated by the subsequent 
transcription. They attempted to elicit counter- arguments from 
interviev/ees who felt the war would be short. As much as possible the 
points at which the counter-arguments were raised were the used 

in the transcription for raising the same counter- arguments . The 
outline of factual materials organized in this way was thus used by the 
script writers as a basis for preparing the program that used arguments 
on "both sides" of the question. It should be noted that, at the time 
of preparing the scripts, the writer knew the purpose of the experiment 
and the actual wording of the main question to be used in measuring the 
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effects of the transcription. 



Among men initially opposed to the position advocated by 
the transcription, the ”one side" message produced a net change in 
the direction advocated by the message of 36%. The "two side 
message (which had been prepared after prior interviewing of the 
intended audience) produced a net change in the desired direction of 
48%. 



In this case then, prior interviewing of an intended 
audience was used to determine the nature of opposing attitudes and 
the stimulus conditions which aroused them. Theoretical 
considerations were then employed to determine an appropriate use for 
this data in constructing an attitude- changing message. 

More recently, Levonian (10) conducted extensive prior 
testing of the opinion dimensions toward India of an intended film 
audience. Prior to scripting the film he derived a number of 
relatively independent opinion dimensions toward India by factor 
analyzing responses to a 69-item opinion questionnaire. Each item in 
the questionnaire (itself a film) consisted of a film scene, a 
narrated question, and an abbreviated repetition of the narrated 
question printed on a mimeographed foirm in the student *s hand. Each 
scene lasted about 15 seconds, during which time the narrated 

question was asked. At the end of each scene a title, ANSWER ITEM , 

appeared on the screen in black letters. The projection- time of the 
title allowed the respondents about 15 seconds to answer each item. 

A factor analysis of the responses revealed ten common 
opinion dimensions, of which eight appeared related to specific 
content areas. These eight opinion dimensions formed the basis for 
the subsequent development of the film. Before constructing the film, 
the researchers decided upon the orientation to be adopted for each • 
content area. This orientation was coincident with the direction of 
the desired change of opinion. 

To test the effects of the final film, Levonian (11) 
selected 540 university students, each of whom was given a 36-item 
questionnaire pertaining to the material presented in the film. One 
experimental group was shown the film a week later and the 
questionnaire was readministered immediately after the film. The 
other experimental group was also shown the film a week later, but 
the questionnaire was readministered one week after the film showing. 
The questionnaire was readministered a week later to the control 
group, which did not see the film at all. Relative to the control 
group, the experimental groups showed large changes of opinion in the 
desired direction. 

Edling (3) conducted an experiment to determine if films 
produced with a knowledge of the learner's motives were more effective 
in changing attitudes among those for whom the materials were intended 
than among individuals in general . Motives were operationally defined 



in terms of responses to existing psychological instruments. A target 
group of high ability high school students who indicated no plans for 
post-high school education were tested on these psychological 
instruments and their responses subjected to a factor analysis. Large 
differences in learner responses by sex, plus the dictates of the 
research design, led to the production of two films, each intended to 
modify attitudes toward post-high school education for a group with 
specific motivational characteristics. It was demonstrated that the 
film intended for the student group with a specific motivational 
pattern mediated greater opinion change for that group than for people 
in general or for groups possessing other motivational pattern 
characteristics. Among other things, he concluded that techniques 
employed by the behavioral scientists can make a significant 
contribution to the development of more effective instructional 
materials at two points: first, in the analysis of learner 
characteristics as a basis for structuring content and, second, in 
empirically testing materials to determine their psychological 
validity. 



In addition to these attitude-changing programs, the 
analysis of student responses to content-related questions has been 
employed to develop programs intended to produce increments of 
cognitive gain. Procedures of this sort are becoming more common, 
especially among developers of self-instructional programs. To a 
limited extent, extensive analysis of data both from pretests and 
from within-program responses is being used as a basis for revision 
by some program developers, (See AERA, 1, Lumsdaine, 1^, and 
Markle, 17 •) 



J. V. Zuckerman (30) pointed out as early as 1951 a high 
correlation (Pearson r's of .71-. 94) between post-test scores obtained 
by one group of subjects which saw a finished color version of a film, 
and the post- test scores of a second group which saw a rough, black- 
and-white filmstrip of the storyboard of the same film. 

The study was concerned with ascertaining experimentally 
the extent to which the responses of typical students to a preliminary 
filmstrip, made from a series of "storyboard" sketches, could serve 
as a predictive device to tell how well students would learn specific 
points of information from a completed film. The sketches were 
intended to represent the most important visual aspects of the scenes 
to be photographed in the motion picture, "Flight Capabilities of ^ 
the F86-A," with the accompanying dialogue or narration indicated for 
each sketch. Two groups of subjects were compared: Group II viewed 
the completed color motion picture. The same sound track was used for 
both groups. Both groups were given pre- and post-tests covering in 
detail the points of information presented in the films. Subjects 
were 90 Air Force student pilots in advanced single-engine pilot 
training. The results, based on percentages of the correct answers 
for each of the 26 multiple-choice test items, were computed in terms 
of several adjusted scores. The level of learning obtained varied 
widely among the various points of information covered by the films. 
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Pearson product-moment correlation coefficients, computed between the 
pairs of item percentages for the two treatments, were .80, .71, .94, 
and .89, respectively, for four measures. 

Regardless of which measures are considered, the varying 
levels of knowledge on specific points obtained from the film are 
thus predicted fairly well by the levels of knowledge obtained from 
the filmstrip. This means that tryout data from the preliminary form 
could have been used profitably for identifying how well various 
points of information would get across and, hence, as a basis for 
knowing on which points it should be strengthened. An iticidental 
finding was that the average level of achievement for the final, 
polished film was roughly the same as for the crude preliminary 
version. (See also May and Lumsdaine, 19.) 

Cropper, Lumsdaine and Shipman (4) made application of 
this finding to demonstrate how such a rough, preview version of a 
lesson might be used as a basis for gathering student response data 
upon which to revise the lesson prior to producing the final version. 

Two different lessons - one on the effects of heat and one 
on an introduction to chemistry - in a junior high school science 
series televised by the Metropolitan Pittsburgh Educational 
Television Stations were shown on separate occasions to a preview 
audience of students. Following the preview showing, students took 
an achievement test based upon the lessons. Analysis of the items 
revealed which points were misunderstood or poorly learned. The 
lessons were then analyzed to discover the possible reasons for their 
ineffectiveness, and were revised to remedy the weaknesses. Later, 
both the preview version and the revised version were telecast 
simultaneously on the day the lesson was regularly scheduled to be 
shown. Students who watched each version then took an identical 
achievement test. 

Results of the experiment showed an average improvement 
varying from 12 to 26 percent for students who watched the revised 
versions over students who watched the original preview versions. It 
was concluded that, as feedback to the programer, test results following 
a preview of a lesson are an effective substitute for the interaction 
between student and teacher that occurs in a classroom. The procedure 
enables the programer to tailor his presentation to the needs and 
abilities of his particular student audience. 

VanderMeer and his associates (28 and 29) report upon a 
similar procedure for using post-test data derived from early versions 
of both filmstrips and films to produce revisions more effective than 
the original. In one case they developed tests based upon the content 
of two existing films and subsequently revised the films on the basis 
of student responses on the tests by adding additional instruction 
and modifying both the graphic and aural content of the films. Two 
successive revisions were made, the second of which produced 
significantly superior performance over the original version. In the 



12 



other case, an existing filmstrip was revised by a similar procedure 
with similar results. 

Markle (18) reports using student performance data to 
support many design decisions in the development of a multi-media 
basic first aid course intended, in seven and one-half hours, to 
produce results at least equivalent to those produced by a standard 
conventional ten-hour course then in current use. The course 
objectives were developed by pretesting a sample of the target 
population to determine levels of entering knowledge. A revised set 
of test questions was then used as a first draft of the course, 
instructional elements being added on each topic until trial students 
could respond successfully to the questions. Each instructional 
element (film, filmstrip, worksheet, or other) was pretested with 
trial students and revised in response to the test results prior to 
final publication .- 

The mean scores of students taking the resulting course 
were as much as 86% higher than those of students who took the older 
standard course, and the time spent ±i the program had been reduced 
by 25%. In the process Markle produced three successive developmental 
versions of the course, the first was a preliminary trial version, 
and each of the succeeding versions was a revision of the one 
preceding it. The methods employed, using student responses to test 
questions based upon the content, showed each succeeding version to 
be more effective than its predecessor in less instructional time. 

Lumsdaine and May (15, p.49A) point out that surprisingly 
few studies have been reported to demonstrate empirically that the 
expected value of successive revision based upon feedback of test data 
to the programer has actually been realized. Earlier versions of a 
revised program are generally not preserved after the revised version 
has been completed. Though the revised version is generally felt to 
be an improvement over its earlier counterpart, this fact has rarely 
been demonstrated. While both Edling (3) and Levonian (10) felt 
that films "tailored" to student characteristics were more effective 
in modifying attitudes than films not so tailored, neither made a 
direct experimental comparison between two such films. In another 
study Holland (6) compared student errors within a program and time 
to completion for both a revised and original version of a self- 
instructional program, but did not report a comparison of post-test 
data derived from students in each of the programs. 

The first such demonstration appears to have been by Cropper, 
Lumsdaine and Shipman (4) who compared simultaneous telecasts of 
earlier and later revisions of a video-taped program and showed 
appreciable gains from both a first and second program revision based 
upon the results of achievement tests. (See also Cropper and 
Lumsdaine, 5.) In later studies, both Silberman (26) and Markle (18) 
replicated these findings in the development, in one case, of self- 
ins tructional programs and, in the other case, a multi-media course of 
instruction. 
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Objectives of the Study 



Three important questions emerge from these lines of inquiry. 
First, given that the framing of items for a test of a program s 
content may be regarded as a process of specifying behavioral 
objectives, we might ask whether an experienced teacher given a clear 
specification of objectives might not predict, a priori , those items 
on which students will likely fail. None of the studies cited above ^ 
compared their objective procedures with such an alternative subjective 

procedure. 

In this regard there is some evidence to suggest that 
teachers might not succeed in this kind of analysis, in spite of the 
popular assumption to the contrary. Rothkopf (23) conducted an 
experiment to determine the ability of "experts" to predict the 
effectiveness of an instructional program by reading it. The actual 
effectiveness of seven versions of a program were previously determined 
experimentally. Twelve educators who had completed a course on 
programing were then asked to read the seven versions and rank them 
according to their predicted effectiveness. The predictions of these 
twelve authoritative persons correlated -.75 (sic) with actual 
effectiveness scores. In other words, a reasonably good prediction 
of effectiveness could be obtained by predicting the reverse of these 
expert predictions (see also Merrill and McAshan, 20) . 

Hovland, et al, (7) relate similar experiences when trying 
to obtain constructive suggestions from viewers on how to improve a 
film’s effectiveness. While readily expressing liking or disliking 
an educational film, they point out, the viewers were singularly 
inarticulate as to their reasons and had few suggestions of ways to 
improve it. Similar findings were reported by Twyford (27) in an 
experiment to determine if a subjective rating profile could be 
considered a reliable measure of learning from a film. None of his 
self-report measures could be used reliably to predict actual learning 

mediated by the film. 

Still the question is an important one from the point of 
view of the cost benefits to be derived from exhaustive testing of 
preliminary versions of an instructional program to produce a more 
effective final product. One of the objectives of the study, 
therefore, was to compare experimentally the relative effectiveness 
of two procedures for revising an instructional program, one based 
upon a subjective review of the program with respect to a set of 
behavioral objectives and the other based upon objective test data 
obtained from an actual tryout of the program. 

A second question to emerge from these studies concerns 
whether certain psychological or demographic characteristics are 
related to the ability to act effectively to improve a program s 
effectiveness. It is clear from most of the studies that the 
experimenters applied objective procedures and produced effective 
programs. The relationship between these two outcomes, however, 
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does not appear to have been assessed. It is not clear, for example, 
if their good results were due to their procedures, per se, or to 
some personal characteristics which enabled them to interact with t e 
data in some unique way. Knowing what the trouble is, even knowing 
where it is, does not ensure that anyone can correct it. None of the 
studies cited formally compared programers on their ability to act 
effectively upon student response data (see Meyers, 21). 

Another objective of the study, therefore, was to relate 
two of the demographic characteristics most commonly felt to affect 
teaching ability — academic preparation and teaching experience 
to the effectiveness of the programer's effort. To some extent it 
should be expected that programers with more education and experience 
in teaching would have a superior basis upon which to prepare a 
supplementary revision than the less experienced programer. The 
a priori analysis of an experienced teacher should predict the actua ^ 
performance of students better than the analysis of a less experienced 
teacher. At least that is the popular notion. Further, given actual 
test data pinpointing specific weaknesses in the program, it should 
be expected that the more experienced programer might act more 
effectively than his less experienced counterpart to overcome those 
weaknesses. In other words, the more experienced teachers should be 
expected to revise programs more effectively than the less experienced 
teachers irrespective of the procedures employed. 

Finally, the most important issue arising from the research 
and the major concern of this study concerned whether a procedure 
for revising a program could be specified which would lead a sample _ q f 
programers to consistently superior results. In the studies cited 
above it was demonstrated only that the experimenters produced 
effective programs. In none of the studies was it formally demonstrated 
that other programers following the same procedures would be likely to 
produce the same effects. Nor was it demonstrated that the use of 
objective student response data was instrumental in obtaining the 
improved effects (see Lumsdaine, 16, p. 292). The major objective of 
the study, therefore, was to compare the effectiveness of two 
alternative programing procedures as they would be employed by a samp e 
of programers to revise an early version of an instructional program. 

These questions were expressed in terms of the following 
experimental hypotheses which were the object of the study: 

Hi : Given a preliminary version of an instructional program 

and a set of its behavioral objectives, programers will 
be able to revise it effectively on the basis of a 
subjective analysis; moreover, those who additionally 
study tryout test data will produce more effective 
revisions than those who do not (original version < 
subjective revision < objective revision). 

H : Amount of teaching experience is positively correlated 

with the effectiveness of the programers' revisions 
(high experience > low experience). 
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Ho: Amount of academic preparation is positively correlated 

with the effectiveness of the programers’ revisions 
(high education > low education). 
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MF.THODS AND PROCEDURES 



Gene ral Design 

An experiment was designed to compare the effects of Wo 
alternative procedures for improving a preliminary version of a- 

irthr^r^UTh-^k-: :Lii:Vth:f --^t-a^hi— 

test data obtained after a trial run of the program. 

Learning tasks were analyzed and a set of behavioral 

ct-ah1i‘?hed for a fixed-pace instructional program 
objectives were J°^„^Jirated with a tape recording on the 

consisting of a set of slides cooraxnai-e _ssgs ^kn achievement 

subject of English Mone^ for use "^3!"®h;n Wials of the 

was developed based upon the stated objectives, tnen irict 

r'iginfl program were condLted using the test to obtain af :er-only 

meaLres of the program's initial effectiveness. 

Twenty programers, selected from a population of state- 
credentialed teachers, were randomly assigned to one of two 
orouos- I (subjective) in which only general statements or objectiv 
^ ^ * -A A nA TT ^obiective) in which achievement test data 

Se “iaf^rwlr provided. Each programer then planned 
and^iideiLped a 15 -minute supplementary lesson based upon his analysis 
of the iformation provided. 

Trials of the 20 resulting videotapes were conducted in 20 

:;srL::“9 - r 

lesson. The effectiveness of each videotaped lesson was represe 
by the class mean of the scores obtained. 

Entire classes, rather than individuals, were assigned at 
random for trials of the videotaped lessons since it was not 
administratively feasible within the framework of «=hoo 

of the area to use f Twd^^t 

classes were sampled, the class ratner cnau 

became the proper unit for statistiLlly treated using the 

ac^nn'flted with each program trial were scdi.it. uj-k-ci y ^ ^ 

class rather than the student as the unit of observation (see Lindquist, 

12 ) in order to determine the relative effectiveness of the treatment 

groups. 

Tne general method of the experiment is diagramed in Figure 3, 
i_ • i-Vi ” T/vr flow of activities involved in the study. General y, 
showing the maj two stages: (1) the development of program 

and^test^mlterials^within the framework of experimentally controlled 
conditions and (2) the trials of the experimentally produced materials 
to obtain measures of their relative effectiveness. 
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FIGURE 3 
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General Design of the Experiment 
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Thus, the general design of the experiment consisted of an 
after-only comparison of the effectiveness of programs prepared by 
two independent samples of programers using different procedures. 

The experimental units of interest were the products of the 
programers and the independent variable was test data provided ^he 
programers in the set of instructions given for them to follow, ^e 
dependent variable was a post- test measure of the effectiveness of 
their programs. In addition, the amounts of education and teaching 
experience were correlated with program effectiveness. 

Definition of "Revision” 

At an early stage a determination of the exact nature of 
the "revised version" had to be made. For prccedural reasons in 
this study, "revision" was defined as "the addition of supplementary 
elements of instruction to a preliminary version of a program." 

This definition was selected rather than one which would permit 
deletions and rearrangements of the base program to simplify both 
the production and presentation of the revised versions. While 
this constraint imposed a somewhat artificial condition upon the work 
of the programer, who developmental ly revises the internal structure 
of a program, the condition was quite analogous to the kind of 
revision practiced by the classroom teacher who is constantly asked 
to provide supplementary instruction to existing published materials. 
It did not seem, therefore, that this constraint would limit the 
generalizability of the findings since it was a revision strategy 
under consideration, not a program form. If one strategy was 
shown to lead programers to superior revision decisions, we would 
expect this to hold true regardless of the forms of the programs 
being revised. On the other hand, this constraint reduced 
considerably the complexity and thereby the cost of producing the 
revised versions. 

The revised version therefore was defined as consisting 
of two elements: (1) the original experimental base program and (2) 
a supplementary sequence of instruction designed to follow the base 



Selection of Media 

The most common revision situation is that which involves 
supplementing an established sequence of instruction, such as a film 
or published book. The choice in this study was to simulate an 
established sequence presented on a fixed-pace basis. The choice of 
a fixed-pace sequence was governed partly by the anticipated need to 
administer the program under time-limited conditions in schools, and 
partly to bring the time-to-completion factor under the control of 

the program. 
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•Die most common fixed-pace medium, of course, is film. 
However, in view of the many similarities between film and a series 
of slides coordinated with an audiotape recording, the far greater 
cost of film production did not seem to be justified. The base 
program, therefore, was planned for presentation by means of an 
integrated slide-tape recording. 

The supplementary elements, however, since they would be 
prepared independently by a number of programers would not lend 
themselves easily to the same medium. The logistics of preparing 
large numbers of slides and tape recordings at the direction of many 
programers seemed prohibitive. Besides, the task involved in 
developing such materials was like?.y to be quite remote from the 
experience of most of the subjects. 

Videotape was selected as the medium for preparing the 
program supplements because, by providing a stand-up teaching 
situation, the tasks required of the subjects would be V7ell within 
the range of experience which might be expected of them, particularly 
if they were drawn from a population of teachers. 

Selection of a Grade Level 



The sixth grade was chosen as the locus for the experiment 
for several reasons. First, since the cost of videotape playback 
limited the number of times each program could be shown, it was 
necessary to plan for group viewing of the experimental revisions. 
Further, while it would have been desirable to randomly assign 
individual students to viewing groups, it was not likely that schools 
would cooperate in such a randomization procedure. More likely, 
playbacks to intact classroom groups would be preferred by schools. 

The departmental organization at the junior high and high 
school levels suggested that classroom groups drawn from that 
population would be grouped by subject. Multiple- tracking practices 
in many schools would make additionally difficult the selection of 
a sample in which all groups had comparable characteristics. Since 
the project was constrained from pursuing a large sample of groups, 
it was necessary to achieve as much comparability among the sample 
groups as possible. It was felt that this would be better achieved 
at the elementary level where students were not grouped by subject 
and where the attempt was often made to avoid ability groupings in 
making up classroom rolls. (These conditions were subsequently 
checked in the selection of the sample classrooms.) 

For ease of administration, too, the elementary level seemed 
suited to the purpose. With students assigned most of the day 
to a single teacher, convenient scheduling of the playbacks promised 
to be more easily achieved than under conditions where the students 
were moving frequently between classes. 
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Sixth grade was selected to achieve the most mature 
elementary students, since the learning activities promised to involve 
both reading and writing and it was desirable to reduce the amount of 
error variance which might be obtained from deficiencies in these skills 



De velopment of the Experimental Bas e Program 

Selection of the Topic — Since the experiment involved 
two procedures for revising a preliminary version of an instructional 
program, it was necessary at an early stage to develop a base program 
which subsequently would be experimentally revised. 



A number of considerations influenced the selection of a 
topic for this base program. First, the experiment would benefit 
from more favorable response from school officials if the total 
program, which was to be administered in schools, could be administered 
inside an hour. Therefore, an after-only design seemed desirable as 
a means to eliminate pre-testing of students within the program. To 
reduce error variance obtaining from differential foreknowledge of 
the topic, therefore, it seemed advisable to select a leai-LiiiLg task 
which could be reasonably assumed to be unfamiliar to the vast 
majority of students. 

With no need to pre-test, the administration of the program 
might be accomplished within the following broad limits: 



Administration of base program 15-20 min. 

Administration of videotape supplement . . . 15-20 min. 

Administration of post-test 10-15 min. 

Introduction, instructions, etc 5-10 min_^ 

Total 45-65 mine 

If the base program were to be administered in 15-20 minutes 
as the above schedule suggested it must, the scope of its objectives 
had to be severely limited. To test the learning of students in the 
short remaining post- test interval, the learning tasks needed to be 
fairly discrete and amenable to efficient, objective measurement. 
Finally, if we wished later to generalize our results across some 
larger population of learning tasks, the experiment would benefit by 
selecting a topic possessing some generalizable traits. 



With these considerations in mind, the topic of EngH^sh 
Money was particularly appealing for the following reasons: 

1. Its learning tasks were largely symbolic and involved 
both verbal and quantitative symbols. 

2. We could reasonably assume not only that the topic itself 
was unfamiliar to most American students, but also that 
its underlying verbal and quantitative systems would be 
equally unfamiliar. Not only would the student have to 
learn new names and associate them with unfamiliar coins, 
but he would also have to learn to calculate values in a 
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non-decimal system, based upon a mixture of 12-base and 
20-base systems. 

3. A large number of unambiguous teaching and testing items 
could be generated from a relatively few principles. 



Once the general topic had been selected, it was necessary 
to limit the scope of the program dealing with it so that 
reasonably be learned within the short time span allowed. 
of the learning tasks was conducted and a set of general objective 
established with this limitation in mind. 

nf the Learning Tasks - The student encountering 
English money for the first time is generally exposed to the coins 
and bills in circulation, to price tags and labels, 
who quote prices and label pieces of money m colloquial speec . 
tasks the student is called upon to perform in the Presence 
aural and visual symbols involve computing equivalences and dif 

within the English money system and expressing the result in term 
, . A typlcal purchase m.ight 

involve a^erchant quoting a price in colloquial speech the Solectron 
of an appropriate set of coins by the student to cover the 
the computation of the correct value of change to be received and the 
assessmLt of equivalence between the value of the expected change ana 
the value of the change received. 

In designing the base program, it was decided to expose the 
student to the expected range of stimulus classes asking h- - -^e 
the typical responses he would be expected to make under realistic 
circumstances. Thus, in the presence of a variety of stimulus 
situations the student would be asked to express equivalences 
differences in value in any of the conventional symbol systems. 

For administrative purposes, since they would be difficult 
to record in the post-test situation, spoken responses were exclude 

from the program. 

The Obiectives of the Program — The following general 
objectives were established for the program (see American Institutes 
for Research, 2, and Orlando, 22). 

1. To be able to recognize and label the seven English coins 
in common circulation. 

2 To be able to calculate equivalences and differences of 
value between coins, written and spoken verbal statemencs 
and numerical expressions interchangeably according to 
the common practice. 

3. To be able to express values in terms of coin selections, 
written verbal statements and numerical expressions 
according to the common practice. 
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specifically excluded objectives were described as follows: 

1. To be able to compute American monetary equivalents of 
British units. 

2. To be able to estimate costs of common items in British 
comnerce. 

3. To be able to demonstrate oral mastery of the content. 

4. To be able to calculate differences involving values in 
excess ot 

Stimulus and Response Classes -- Since it would be difficult 
to provide all students with a set of actual British coins, it was 
decided that the program and subsequent test would present photographic 
representations of the coins and their respective labels as follows: 
half-penny, penny, three-penny bit, six-penny bit, shilling, florin, 
and half-crown. The program would present a range of values from ^d. 
to several pounds expressed in terms of written words, spoken words and 
numerical expressions. In addition, values from kd. to 19/11% would 
be presented in the form of photographic representations of coins. 

Tlie student would be asked to respond to these stimuli by 
0xpressing equivalences and differences of value in the form of written 
verbal statements, numerical expressions, and selections from among 
alternative sets of coins represented photographically. 

Table 1 shows the stimulus and response elements as they were 
operationally defined in the program. 

The Base Program as a "Preliminary Draft” — An initial draft 
of the program was written and recorded on audiotape to determine the 
elapsed time from start to finish. The first version ran over 40 
minutes and was subsequently cut to 25 minutes by deleting and 
collapsing frames. As it was the intent of the base program to 
represent a preliminary draft version, no prior testing was conducted 
with students and much of the instruction which was felt to be 
ineffective was left to stand. The final version of the program ran 
25 minutes and consisted of 65 2- by 2-inch slides coordinated with a 
tape (See Appendix B , p. 71). 

Development of P ost-Test 

Since the post- test was to assess in part the student's 
ability to recognize oral expressions and statements, it was designed to 
include both visual and aural stimuli. Thus, a tape recording was 
developed to accompany a paper-and-pencil test. 



The technique of playing a tape recording as part of an 
achievement test was useful for other purposes as well. By this means 
it was possible to standardize not only the aural elements of the test 
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TABLE 1 



Operational Definitions of Stimulus-Response Elements 
Managed in the Base Program 



Program Element 


As Stimulus 


As Response 


Coin Configuration 


A photograph of coins 
is used as a stimulus 
pattern. 


S selects from several 
alternative photos of 
coins to express 
equivalence or 
difference. 


Coin Labels, 
spoken 


The names of coins 
are spoken colloqui- 
ally to ^ (eg. 
"Thrup'ny Bit")* 


^ speaks the names of 
coins colloquially. 
However, spoken re- 
sponses are excluded 
from this program. 


Coin Labels, 
written 


The names of coins 
are written out 
correctly spelled 
(ie. not as colloqui- 
ally pronounced, eg. 
"Three-penny Bit") . 


^ writes the names of 
coins as they would be 
correctly spelled 
(ie. not as colloqui- 
ally pronounced, 
thoiigh spelling is 
NOT an objective of 
the program) . 


Value Statements, 
spoken 


A value is spoken to 
S colloquially (eg. 
"Tuppence Hayp^ny"). 


^ expresses a value 
colloquially. However, 
spoken responses are 
excluded from this 
program. 


Value Statements, 
written 


A value is written 
in words (eg. "Two 
Pence Half~Penny") . 


^ expresses a value 
in words by writing 
them as a response 
(eg. "Two Shillings"). 


Value Expressions 


A value is expressed 
in terms of numerical 
symbols according to 
the conventional 
practice (eg. "12s. ld= 
12/1"). 


^ expresses a value 
in numerical symbols 
according to the 
conventional practice 
(eg. "6/5=6s.5d."). 
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but also the timing of the test. Further, it was decided to use the 
recording to read ^ the test items as a means of reducing the e 
variance likely to accrue from expected reading deficiencies at the 

sixth-grade level. 

Thirty test items were developed and recorded using a 
representative sample of the item-generating statements shown in Table 
2 A trial of the base program was conducted in two randomly selec 
sixth-grade classes and Che 30 items were administered as a test 
following the program. Student responses to these items were analyzed 
by computing indexes of difficulty and discrimination for each item. 

The most difficult and least discriminating items were thus 
eliminated in developing the final ten-item test. 

The initial 30-item sample is shown in Appendix A, p. 57, 
together with the indexes of difficulty and discrimination obtained 
for each item. The final version of the post-test appears in Appendix 

B, p. 82. 

Standardization of Treatments 

The task assigned to all the programer-sub jects (jj) in the 
experiment, was within a single day, to examine a preliminary version 
of a 25-minuCe lesson on English Money, consisting of a sec of 2- by 
2-inch slides coordinated with a cape recording, Chen co^review a sec 
of instructions which included statements of Che program s objective, 
and finally Co plan and record on videotape a 15-minuCe supplementary 
!esson calLlated to help sixth-graders score well on a post-test 
following the program. 

To assure chat comparable conditions prevailed for all S’s, 
standard procedures were followed in recruiting subjects for "e 
experiment, introducing them to their task producing ‘heir videotaped 
supplementary lessons, and administering these programs in classrooms. 

Recruitment — The following advertisement was placed in a 
local newspaper: 

Teachers needed for TV teaching project on UCSC campus. 

One day only. Call . . . 

During the subsequent telephone interviews, the experimenter 
(E) determined if the callers possessed the minimum qualifications for 

?nfo^^?ly, the project was described in the following standardized 
terms : 

We are engaged in a study concerned with television teaching, 

particularly with Che kinds of information teachers need to 

help them teach effectively by television. 
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TABLE 2 



Behavioral Objectives Expressed 
as Test-Generating Statements 



S is given any of the 
following: 


And is asked to 
subtract any of 
the following: 


And to express his 
answer by any of the 
following: 


- The names of some 
coins (written out) . 


- The value of some 
coins named in 
writing. 


- Naming a correct 
set of coins in 
writing. 


- The names of some 
coins (spoken 
aloud) . 


- A given amount 
of money spoken 
aloud. 


- Stating in words 
(written) a correct 
amount of money . 


- A set of coins 
(in photographic 
form) . 


- A given amount 
of money written 
out in words. 


- Expressing a correct 
amount of money by 
writing a conven- 
tional numerical 
expression. 


- An amount of money 
(written as a 
conventional 
numerical expression) . 


- An amount of money 
written as a con- 
ventional numerical 
expression. 


- Selecting a correct 
set of coins from 
several sets shown 
in photographs. 


- An amount of money 
(written in words) . 


- The value of a 
given set of coins 
(in photographic 
form) . 


(Oral responses 
excluded) 


- An amount of money 
(spoken aloud) . 


- The value of some 
coins named aloud. 





If you are selected as one of the participating teachers, 
this is what you would do: we would set up a time when it 

was convenient for you to spend a day with us at our studios 
on campus. At that time you would be assigned a topic and 
asked to prepare a 15-rainute lesson on that topic which 
would be presented in the studio and recorded on videotape, 
all on that same day. Later we would play back your videotape 
in one or two schools in this area to determine how well 
students learned from it. 

When you arrive here, we will assign you a topic and then 
show you around the studio and demonstrate the tools and 
techniques you can use to teach it. You will have most of 
the day to plan your lesson and try it out before recording 
it. My staff and I will be there to help you at all times. 

Most of our participants have had no prior experience with 
television, so we are prepared to teach you the basic 
fundamentals. Also, since this is a recording, you can 
prepare one smalJ part at a time, review it, and then do 
it over as many times as you wish until you are satisfied. 

This is basically a research project, so no publication of 
the videotape is planned. If you are interested in 
television teaching, this is a good way to start. If you 
decide you would like to join with us, the project is 
prepared to compensate you $20.00 for your participation. 

Introduction to the Programing Task — It was anticipated 
that most subjects would have reason for feeling insecure when faced 
with the task that was proposed. Few had any experience with 
television in the first place, which accounted for their initial 

t in the project. Experience had indicated, however, that the 
first exposure to the hardware and staff associated with television 
would arouse feelings of insecurity. Further, for most subjects, the 
topic of English Money was certain to arouse additional feelings of 
tension -- it was a topic calculated to be unfamiliar to American 
audiences. The introduction, therefore, while concerned with 
standardizing the experimental conditions for all subjects, was equally 
concerned with relaxing each subject and providing a basis for feeling 
confident. The following discourse was used as a guide, therefore, 
and every opportunity to gain the subject's participation in a dialogue 
was exploited: 

We are preparing a lesson for sixth-graders on the topic of 
English Mo ney . The entire lesson will consist of several 
7lements. First, we have prepared a 25-minute sequence of 
slides with an accompanying tape which introduces the subject. 
It is written after the fashion of self- instructional 
programs, so the students are given a chance to practice 
what they learn as they go along. You will have a chance 
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to see this sequence, and go through it just as the students 
will. That way you will see exactly what the students will 
see and, at the same time, learn about English Money 
yourself. 

The second part of the lesson w-ill be a 15-minute videotape 
supplement to the slide sequence, which you will prepare 
today. The object of this videotape is to provide whatever 
additional instruction or practice you think the students 
will need to help them pass a test on the subject. 

Finally, the students will be given a brief test on the 
subject to see how much they have been able to learn from 
the entire lesson -- the slide sequence, and your videotape 
supplement . 

We have prepared this booklet of materials [see Appendix B, 
p. 65] to help you in your task. It contains, here, a general 
description of the objectives which the lessor, will try to 
achieve. Here is a table which defines the stimulus-response 
terms operationally. Next, here is what might be termed a 
"test-item generator." This table shows what kinds of test 
items will appear in the final test of the subject. Following, 
here, is a copy of the script which was used for the slide- 
tape sequence so you can review it easily. [Treatment II 
adds: Finally, he 2 is a copy of the specific items used 

in the post- test together with data showing how well sixth- 
graders perform on each item after seeing just the slide-tape 
sequence.] 

Your job is simply this: study the slide-tape sequence and 

the objectives [Treatment II adds: . . . and the test 

results] . On the basis of your own experience and 
knowledge about how people learn, try to plan a lesson which 
will help the students pass the final post-test. [Treatment 
II adds: The only thing we must ask, of course, is that you 
do not use any of these specific test items in your 
presentation.] You will have six hours in which to complete 
your tape. You should plan to spend no more than three hours 
in planning before you start to practice in the studio. 

At this point, ^ was given a tour of the studio which 
demonstrated the studio facilities and the operations of which they 
were capable. In particular, the capabilities of the zoom lenses were 
demonstrated as well as applications of the overhead camera. It was 
pointed out that their lesson could be expository or it could involve 
student practice as they wished. 

Following the introduction to the studio, ^ was seated in the 
planning area and the slide- tape sequence was started. From then, S 
was on his own to plan his lesson. The technical staff remained in 



the area, out of sight in the control room, to provide technical 
assistance in the studio if ^ wished it. 

The Physical Arrangements -- A standard studio setup v;as 
provided to all S's and all were limited to using only the resources 
provided in the studio. The standard studio setup, shown in Figure 4, 
provided the following complement of facilities: 

1. A fixed camera with 15-150 ram. zoom lens 

2. A movable camera with oper.tor and 15-150 ram. zoom lens 

3. A subject-operated overhead camera system with 15-150 ram. 
zoom lens 

4. Two studio monitors 

5. A chalkboard 

6. A world map 

7. A globe 

8 . Miscellaneous furniture as needed. 

A separate area of the studio was set up with a desk and 
r- 0 ar-pro j ection slide viewer for displaying the base program and 
planning the supplementary lesson and graphic materials, shown in 
Figure 5. The total complement of items provided to ^ in this area 
consisted of the following: 

1. An instruction booklet for either Treatment I or 
Treatment II (see Appendix B, page 65.) 

2. A set of English coins (one half-crown, two florins, 
three shillings, five six-penny bits, four three-penny 
bits, seven pennies, and four half-pennies). 

3. Blank paper and felt- tipped pens. 

General Constraints -- Several constraints were imposed upon 
all these activities to assure comparability of treatments. First, no 
S was informed of the topic to be taught at any time prior to his 
arrival at the studio on the day of production. In this way, E could 
assure that ^'s were not able to prepare for the production in any 
differential ways. Particularly, the possibility that S's with later 
appointments would gain an advantage by being able to prepare longer 
was avoided. 
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FIGURE 4 







FIGURE 






Further, technical studio personnel were constrained from 
offering suggestions or interacting with the ^'s in any other than 
technical matters. Consequently, there v;as no "directing*' of the 
subjects in the usual production sense, nor was there interaction with 
£ which might bias strategic decisions in planning the lesson. The _S's 
were left totally responsible to plan the content of their lessons and 
the technical staff instructed to confine their assistance to technical 
matters entirely. 

Finally, the exact nature of the variables under 
consideration was withheld from the subjects as well as from the 
technical staff throughout the experiment. In this way, whatever 
effects might result from the interaction of the _S*s with the 
television staff might be randomly distributed across treatments. 

From the time of arrival until final completion of the 
videotaped lesson all ^'s conducted their work entirely within the 
immediate studio area without leaving the premises. 

Introduction to the Classroom Trials — A standard procedure 
was followed in the trial classes to assure that all programs were 
administered under comparable conditions. 

Prior to the trial, it should be noted that no teachers 
were advised of the subject of the program so that all classes first 
learned of the topic as presentation of the lesson was started. School 
administrators had to be informed of the topic in advance, however, 
in order to gain their approval for presenting the program in the 
schools. In all cases, the importance of avoiding prior preparation 
on the subject in the trial classes was emphasized. The procedure 
followed by them was to advise the teachers only that a "worthwhile 
subject** would be taught. In every case the possibility of prior 
preparation was examined in the introductory procedures to assure 
comparability in this area. In no case was prior preparation in 
evidence. 



Tlie following brief introduction was given to each class 
immediately prior to starting the lesson: 

Hello, boys and girls. My name is and I work 

for the University of California. I have a special lesson 
for you today which is such a surprise, even your teacher 
doesn*t know what it is about. 

Vie are making a filmstrip ... is there anyone here who 
doesn't know what a filmstrip is? Right, it*s a series of 
pictures that teach a lesson, isn’t it? . . . Well, we are 
making a filmstrip on the subject of ^giisl^Mqn^ and you 
are going to help us find out how good a job of teaching 
our filmstrip is doing. 
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Some of you may already know something about English money. 
Who remembers why English money was in the news recently? 
(Field responses which may cover devaluation of the pound, 
and which may name some of the denominations of English 
money. Make sure prior knowledge is limited.) Good. Well, 
today we are going to learn about the denominations of 
English money, the names of their coins and how to make 
change in the English system. 

Now this is a different kind of a lesson. This filmstrip 
has a tape that goes along with it, so you not only watch 
the pictures, but you listen to the teacher as well. The 
teacher will ask you some questions as you go along, and you 
should try to write your answers to these questions on this 
answer sheet. It will give you practice in what you are 
learning. (Pass out worksheets.) Let's look at this 
worksheet. You can see there are a lot of empty boxes on 
the right side of the sheet, numbered from 1 to 35. And 
there is one marked "A" just before the first box. That one 
is for practice. During the lesson, as the teacher asks you 
questions you are to try to think of an answer to each 
question and write it down in the answer space. Remember, 
this is just a worksheet, not a test. We're going to throw 
these away afterwards. Notice the scratch area on the left. 
That is for the arithmetic you might have to do in figuring 
up your answers . 

Right after you have seen the filmstrip, we have prepared a 
review lesson on television. So you will have a chance to 
review what you have learned. [See note below.] 

Finally, after the reviex7 lesson, we will have a short test. 

But remember, this is not a test to see how well you do. 

It's to see how well 1 have done at teaching you. If I have 
done a good job of teaching, then I'll know this filmstrip is 
a good one because you learn from it. But if you don't 
learn too well, then I know I haven't done a very good job 
and I will take the filmstrip back to the university and 
try to make it better. 

In all the things we will do in the next hour, you will not 
put your names on anything. This will not be part of your 
grade. So you can just relax, and enjoy yourselves, and 
see how much you can learn about English money. 

To review: First, the filmstrip lesson with the worksheet, 

then the television review, then a test. Any questions? 
(Answer procedural questions.) Then let's start the tape. 

If you need a pencil, raise your hand. Otherwise, once we 
start, we cannot stop. Do your best, and try to keep up 
with the lesson at all times. Here we go. (Start base 
program. ) 
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(After base program.) 

All right, now turn your worksheet over so you have a blank 
sheet of scratch paper for the television review. Try to 
fix in your mind what you need to know to answer the 
questions on the final test. (Start videotape program.) 

(After videotape program.: Collect all worksheets, pass out 

test booklets.) 

Remove the blank cover sheet on the test booklet. Use this 
to ^over your answers (demonstrate) and for scratch paper if 
you need it. Let's look at the test. These empty answer 
spaces on the right are for your answers. Please put them 
there so I won't miss any when I correct the papers. 

I have prepared a tape to go along with this test. Each 
question is read to you, and then a certain amount of time 
is given for you to answer the question before going on to 
the next question. Do the best you can, and try to keep up 
with the tape. You can go back if you want to, but when t 
tape is over, the test will be over. Questions? (Field 
procedural questions only.) (Start tape.) 

(After test, collect papers; thank all.) 

Note: All references to the television review were omitted for the 

base program trials. 

Administration of Program Trials 

As supplements to the base program, 20 videotaped lessons 
were prepared by 20 programers representing two independent samp e 
groups following different procedures. Representative scenes from 

these programs are shown in Figures 6-25. To assess 
effectiveness of each procedural group, measures of effectivenes 
were needed for the videotaped lessons produced by the programers 

assigned to each group. 

The strategy employed called for a trial of each videotaped 
lesson to be conducted in one randomly assigned, intact class. T e 
post-Lst was to be administered following the lesson and the mean of 
rhp scores obtained by that class, expressed as a percentage o 
pLsible sco^e! was to serve as a measure of the lesson's ef fectrveness 

The 20 trials of the videotaped lessons were conducted in 

20 randomly assigned sixth-grade classes 
In addition, during this same period seven trials of 

without supplements, were conducted in seven randomly assigned sixth 
grade classes. 
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ANALYSIS AND FINDINGS 



To determine the relative effectiveness of the original 
version of the program (orig) and the revisions produced under two 
treatment conditions (revi and revll), the results of an identical 
test administered to all trial classes were analyzed. The means (M) 
of the class means for all treatments were analyzed to test the 
significance of their occurrence in a predicted order of magnitude 
under Hq, i.e., to test Hq: ^orig ~ ^revl ~ ^evll specific 

alternative < ^evll* 

Problems of Analysis 

The experimenter was able to state, at the outset, the 
expected rank order of magnitude of the effects of the programs prepared 
under the different experimental treatments. It was expected that, of 
all the programs, the original, unrevised version would occasion the 
lowest achievement scores, that the revised version prepared by 
subjective analysis of behavioral objectives (revI) would, as a group, 
occasion higher scores, and that the revised version prepared by 
objective analysis of test scores obtained after a trial run of the 
original (revll) would, as a group, occasion the highest scores. A test 
of significance was desired especially sensitive to those differences 
which, while tending to reject the null, at the same time would lend 
support to the specific alternative Hi, that the magnitudes of the 
treatment effects would occur in a particular order. 

It is known tiiat the customary one-way analysis of variance 
does not satisfy this demand: the F-ratio, as well as the H-ratio and 

other k-sample statistics are independent of the order in which the 
sample means occur. Furthermore, attempts to assess the joint 
probabilities yielded by F and some coefficient of rank correlation 
between an expected and an observed order of means meet with a number 
of difficulties (see Jonckheere, 8). 

Thus we had the problem of determining if the results obtained 
from a one-way analysis of variance design supported an alternative 
hypothesis that the magnitude of the effects produced by the different 
treatments occurred in the predicted order. When only two treatments 
are under consideration, the hypothesis that the means will occur in a 
specified order is taken into account by the usual application of the 
one-tail test of significance. If E is unconcerned with the direction 
of difference, the two-tail test is used. A procedure which extends the 
notion of a predicted ordered alternative to the k-sample case is not in 
common use. 

Jonckheere 's Procedure 



Jonckheere (8 ) proposes a procedure whereby the statistic 

S may be used for testing the null hypothesis against the alternative of 
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ordered cumulative distribution functions according to the following: 



k-1 k 



k-1 k 



^ 2 Pij ■ 2 2 

i=l j=l+i i=l j=l+i 



( 1 ) 



^^here c- , is obtained bv arrangine the samoles in the order implied by 
and then, for each sample in turn, determining for each value the 
number of items which are larger in all succeeding samples. 

The value of S yielded by this procedure he shows to be 
precisely the same as that obtained in the calculation of Kendall's 
S between two rankings, ti7hen one of them contains ties. In view of 
the identity of the statistic employed in this test and Kendall's S, 
Jonckheere shows that for large samples 



S-1 



z 



_L.|n^(2n+3) 

18 ^ 



X]mr(2mr+3^ 



( 2 ) 



is approximately normally distributed with zero mean and unit variance. 
Therefore, the probability associated with the occurrence under Hq of 
any observed value of S may be determined by computing the associated 
value of z and then determining the significance of that z by reference 
to a standard normal curve table showing the probabilities associated 
t^ith various values of z. 

It was proposed then to test Hq: Morig = ^revl “ ^revll 
against the specific alternative < Mj-gvl ^ ^revll 

calculating S as shown by Jonckheere and then the associated value 
of z, testing the significance of the observed value of z. 

Analysis of Data: Hq vs. H]^ 



The data we wish to analyze are shown in Table 3. It will be 
noted that the samples are arranged in the order of predicted 
increasing magnitude and that the means are, in fact, in the 
predicted order. We wish to test the hypothesis that the three 
samples have come from the same population against the alternative 
that the populations arc such Lhac the values of the three samples 
are in the expected order of increasing value. 

For the computation of S we have 



m]^=7, m2=ll, m3=9, k=3, 
P12=54, P13=55, P23=78 
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Hence, using (1) we have 



S = 2(54 + 55 + 78) - 239 
= 135 



Now computing the associated z using (2) we have 



z 



135 - 1 



/yjX|(27)2(57)] - [(7)2(17)+(11)2(25)+(9)2(21)] 



134 

44.72 



= 2.996 



By reference to a normal distribution table we find that z > 2.996 
has a probability of occurrence under Hq of p < .0013. Thus, we 
reject Hn that the samples were drawn from the same population and 
accept the specific alternative Morig ^evl ^evll» t at t e 

samples were drawn from populations with ordered emulative 
distribution functions as expected at a level of significance o 
p < .0013. This analysis is summarized in Table 4. 

The cumulative distribution characteristics tested by this 
procedure are shown graphically in Figure 26. Reference to the 
figure shows various values of the cumulated sums of class means 
arrayed along the axis of ordinates. Along the axis of abscissas 
are arrayed the various ranks by which any given class mean may be 
identified within its own sample. Reference to Table 3 shows the 
cumulated sums of class means relative to the within-sample ranks. 

For example, to plot the point associated with the lowest class mean 
(rank 1) in sampleQ^ig? plot (1, 17.78). The next point in 
sample • (rank 2) adds the class mean 18.65 to arrive at the 
cumulated^sum at rank 2 of 36.43. The second point is thus plotted 
(2, 36.43). By this process the cumulative distribution 
characteristics of the class means are plotted for each sample group. 
It can be seen that these functions occur in the order of magnitudes 
predicted under Hj^. 
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Summary of Ranked Class Means From Trials by Treatment 
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’*Each of 27 programs was "randomly assigned to an intact class for trial. Each datum above consists 
of the mean of percentage scores obtained by that class on the post-test administered after the 
program. Accordingly, the data were statistically treated using the class rather than the individual 
student as the unit of observation, with the class mean serving as the measure of class performance. 
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FIGURE 26 



Cumulative Distributions of Class Means by Treatment* 
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To confirm this finding in the more traditional fashion, 
the Kruskal-Wallis one-way analysis of variance by ranks procedure 
(Siegel, 25, 184ff.) was applied to test Hq: Morig'^MrevI^MrevII 
against the more general alternative that Morig ^ MrevI r MrevII 
yielding H = 15.03 with df = 2. The probability associated with this 
value of H allows rejection of the null at p<.001 and acceptance of 
the hypothesis that the samples were drawn from different populations. 

Purther two— sample comparisons were performed using the 
Mann-Wtiitney U Test (Siegel, 25, p. 116ff.) to assess the reliability 
of the differences. With regard to the hypothesis that Morig"^ 
MrevI+revII it was found that the revised versions, irrespective of 
which procedure was used, were reliably more effective than the 
original version at a. one-tail level of significance of p< .025. 
Furthermore, with regard to the hypothesis that MrevI^ MrevII’ the 
major hypothesis dealt with in the experiment, it was found that the 
revisions produced following the objective procedure were reliably 
more effective than the "subjective" revisions at a one-tail level of 
significance of p< .025. 

When we test the hypothesis that Morig*^ Mrevl> however, 
our results are not nearly so conclusive. Here we find, while the 
obtained difference occurred in the predicted direction, we are 
unable to rule out that this difference resulted from chance 
fluctuations. Thus, if we examine only the data deriving from tests 
of the original program and the "subjective" revision, we are unable 
to reject Hq: Morig = MrevI* It is clear from this analysis, as it 
is graphically in Figure 26, above, that the bulk of the variance 
obtaining from tests of all versions of the program can be accounted 
for by the difference between the effects of revision II and the 
effects of all other versions. 

Accordingly, the hypothesis that MrevII = Morig+revI was 
tested against the alternative that MrevII ^orig+revl* was 

found that the "objective" revisions were reliably more effective 
than both the original versions and the "subjective" revisions, 
taken as a group, at a one-tail level of significance of p< .01. 

To review, with respect to the major experimental hypotheses 
tested in the experiment, it was found that the effects obtained from 
trials of the original program and the experimental revisions occurred 
reliably in the expected order of magnitude. The absolute magnitudes 
of the differences obtained were such that a more reliable 
difference was found between the effects of the "objective" revisions 
and the "subjective” revisions than between the "subjective" revisions 
and the original version when these differences were analyzed 
separately. Yet, when taken together, support was found for the 
hypothesis that both revision procedures would improve upon the 
original, the "objective" procedure to a greater extent than the 
"subjective" one. 



Analysis of Data: Hq vs. H2 and H3 



Additional interests in the study were concerned with the 
association between program effectiveness and certain programer 
variables. Under H2 and H3 we expect to find a positive correlation 
between the effectiveness of the produced program and both the 
academic preparation and teaching experience of the programer. 

The data obtained for this analysis, in addition to the 
class means of post-test scores, were self-reports by the programers 
of their prior training and experience. The statistic used herein to 
measure this association is the Spearman rink correlation coefficient, 
rho, here represented by rg. 

The data for the calculation of rs is reported in Table 6 
with the rav7 data already converted to ranks. Correlations between 
program effects and each of the programer variables are reported 
separately for each treatment group in Table 5 . 

Reference to a standard table of critical values of r^ shows 
none of the observed correlations to be reliable at a level of 
significance of p < .05. Thus we cannot reject Hq that the observed 
values are the result of chance fluctuations. 

Summary of Findings 



Two samples of programers revised a preliminary "draft" 
version of an audio-visual instructional program by preparing a 
15-minute videotaped supplementary lesson. The programers in one sample 
based their revisions on a subjective analysis of the program's 
objectives; those in the other, on test data obtained after a trial 
run of the original. All versions of the program and the post-test 
were then administered in sixth-grade classes. 

The effects produced by the programs were found to occur in 
the predicted order of magnitude at a significance level of p <.0013 
the original program was the least effective; the subjective revisions, 
as a group, were more effective; and the objective revisions, as a 
group, were most effective of all. 

No significant correlations were found between program 
effectiveness and the amount of either academic training or teaching 
experience possessed by the programer. 
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CONCLUS IONS AUD RECOMMENDATIONS 



One of the important outcomes of educational research is the 
evolution of a body of tested methods and procedures for developing 
effective instructional products and practices. The underlying 

p]_0S revealed by basic research which help us to understand the 
instructional process are only useful to the extent that they lead 
directly to more effective instructional methods. Often we learn of 
principles and procedures which have been applied effectively by others 
only to find that we do not obtain the same beneficial results when we 
try them for ourselves. 



The object of this study was to try out an experimental design 
for comparing alternative instructional procedures which would provide 
evidence that instructors other than the experimenter, by following a 
specific procedure, could improve the effectiveness of their 
instructional materials. Therefore, the design called for a number of 
programers to try alternative procedures to determine if one of the 
procedures led to the production of more effective materials than the 

other. 



The alternative programing procedures selected for a test of 
this model concerned two analysis procedures in common use^ for analyzing 
and revising an instructional program! one based upon an expert 
subjective analysis of the learning objectives; the other, upon an 
objective analysis of test data obtained from a trial run of the 
existing program. It is a tradition of long standing among educational 
publishers and producers to seek the review of "experts" during the 
development of instructional products intended for publication or mass 
distribution. Typically, reviews by experts are conducted while the 
materials are under development in order that the final product may be 
as effective as possible. On the other hand, during the past decade, 
arising from techniques developed within the programed learning 
movement, more empirical analysis procedures have been advocated. These 
procedures involve testing the developmental materials with 
representative samples of the intended learner population and analyzing 
the results of tests based upon the program’s learning objectives. 



If we are to refine the technology of program development, if 
we are to develop a body of increasingly reliable methods and 
procedures for improving instructional effectiveness, it seems 
reasonable that we should seek to specify not only what must be done, 
but also who must do it. Little has been done, however, to determine 
if the ability to conduct these kinds of analyses and to use the 
findings effectively are common traits among people in general or if 
there are specific psychological or demographic characteristics 
associated with these abilities. It is possible, for example, that the 
expert review may constitute an effective developmental procedure i we 
become very expert in our selection of "experts." It is also possible 
that the analysis of data derived from pre-testing may be a more 



effectiVG proc6dur6 for souig prograinors tilisn for otliors snd W6 may 
profit from becoming more expert in our selection of programers. 

The model tested in this study, therefore, compared the 
effects of two competing programing procedures upon the products of 
various programers who applied them. A number of programers then 
followed one or the other procedure in developing an audio-visual 
instructional program. The effectiveness of the resulting programs 
was then analyzed to (1) compare the effectiveness of the procedures, 
and (2) to relate the effectiveness of the resulting programs to 
certain demographic traits of the programers. 

It was found that the analysis of test data derived from a 
tryout of the original program was reliably more effective than the 
analysis of learning objectives alone when a number of programers used 
these procedures as a basis for improving a program. However, a 
correlational analysis failed to show that either the amount of 
academic training or the amount of prior teaching experience (both 
commonly assumed to be related to instructional effectiveness) were 
reliably related to the effectiveness of the programs . 

It should be noted, however, that the negative correlation 
found between the effectiveness of programs revised subjectively and 
the amount of the programers' teaching experience was of such 
magnitude as would have been reliable at a one-tail level of 
significance of .05<p<.10 had this been the predicted direction of 
the relationship. An outcome of this sort would tend to lend support 
to the findings of earlier research (see Rothkopf, 23, and Merrill 
and McAshan, 20) which showed high negative correlations be^een 
expert predictions and objective measures of program effectiveness. 

If it should turn out that the subjective analysis of experts 
consistently leads to erroneous predictions , it is possible that 
revisions based upon such predictions may actually diminish the 
effects of the original program. While this outcome in the present 
study can be explained as a chance fluctuation, both the magnitude and 
direction of the correlation surely raise questions worthy of further 
inves tigation . 

It would appear then that the experimental model employed in 
the study offers a viable procedure for assessing the relative 
effectiveness of alternative programing procedures as they are applied 
by a sample of programers. The analysis of student test data has been 
widely advocated as an effective revise an instructional 

sequence and the results of this study tend to support this claim. We 
are able to conclude that a population of state-credentialed teachers, 
at least, seems reliably able to make effective use of objective test 
information to improve their instruction. 

Clearly, similar experiments might be conducted to test the 
effects of other programing procedures. Competing theoretical 
learning principles, for example, might be translated into competing 



sets of programing procedures which could then be tested in this 
manner. The results of such experiments would contribute to the 
development of a technology of education in two ways:^ first, by 
providing empirical support for the theoretical position and, second, 
by specifying a reliable procedure for applying laboratory theory to 
a practical teaching problem. 

One of the factors inhibiting the development of a 
reliable technology of instruction concerns our inability at this time 
to clearly describe whether the effects of a procedure justify its 
cost. While most current research, as Lumsdaine and others have 
repeatedly pointed out (4, 15, 16, 19), reports the reliability of the 
observed effects, very little reports upon the amount of the effect in 
terms which are useful for making decisions to use or not to use the 
procedure which produced it. Markle (18), too, has pointed out that 
■the typical developmental study fails to generate data regarding the 
cost-effectiveness of the procedures used as they might compare to 
some alternative procedures. Any attempt to assess the cost- 
effectiveness of programing procedures leads to the inevitable 
problem of quantifying both the learning benefits and the costs of 
achieving them. At present the parameters of these variables are too 
little known to develop a satisfying cost-benefit model which allows 
a rational decision-making analysis in educational contexts. 

In training contexts, on the other hand, (as the distinction 
is commonly employed) we are provided classic models for cost-benefit 
analysis. Markle, for example, may not have provided an experimental 
comparison of his procedures with some alternative procedures; 
nevertheless, the cost-benefit of his effort may be fully assessed. 

His objective was to realize student-time savings against a standard 
test of specific acquired knowledge and skill in a context where 
student-time was valued in dollars. The cost of the effort may be 
traded off directly against the value of student-time to determine 
the cost-benefit derived. In the educational context, however, the 
issues are not nearly so clearly defined. Student-time is not valued 
in dollars, nor are there widely accepted standards for valuing 
student learnings. In the present study, for example, we were able to 
show only that with additional efforts we reliably achieved additional 
effects; moreover, the procedure which involved the most additional 
0ffort reliably produced the most additional effect. Whether the 
effects of the procedures are worth the efforts needed to bring them 
about cannot as yet be answered. How, then, are we to develop 
appropriate cost-benefit models for the instructional procedures we 
employ in educational contexts? 

One model that might be suggested would describe learning 
effects as a function of costs, rather than as a function of learner 
time. After all, school operating costs are valued in dollars which 
provides at least one of the relevant parameters for a satisfying 
cost-benefit analysis. In any case, if there is to develop an 
effective educational technology concerned with applying knowledge of 
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the basic sciences to the practical arts of teaching, educational 
research models are needed which address themselves to the family of 
problems associated with teaching more students more effectively while 
holding the cost-basis relatively constant, t^hile this pro em was 
not dealt with directly in the foregoing experiment, it is worthy ot 
note that the proposed model in addition to testing the relative 
effectiveness of two procedures would additionally provide a basis 
analyzing the relative cost— benefits. 
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AUDIO SCRIPT TO ACC0>1PAIJY TRYOUT TEST 



36. In words, write the total value of these coins. 

37. In words, write the total value of the coins named here. 

38. In symbols, write the expression for two pounds, six shillings, 
fourpence, hay— p^ny. Listen again. Two pounds, six shillings, 
fourpence, hay-p’ny. 

39. In words, write the value of a florin. 

40. With these coins you pay two-and-nine . The clerk hands you one 
coin in change. What is the ncime of that coin? 

41. To pay one-and-three you hand the clerk a half-crown. In words, 
write the value of your change. 

42. You owe a friend six shillings ninepence. You give him these 
coins as part payment. What single coin is still needed to pay 
him in full? 

43. From the total amount shown here, you spend the coins shown here. 
In symbols write the amount you have left. 

44. You have four shillings sixpence in your pocket and you spend 
half-a-crown. What is the name of the single coin you have left? 

45. From the amount shown here you spend a florin and a tnrup ny bit. 
Name the two coins you still have left. 

46. From the amount shown here you spend the coins shown below. In 
S3rmbols, write the amount you have left. 

47. You have ten shillings. You spend six shillings, thrup nee, 
hay-p'ny. In symbols, write the amount you have left. 

48. You have one pound. You spend the amount shown here. In symbols, 
write the amount you have left. 

49. From the amount shown above, you spend the amount shown below. 

Name the two coins you still have left. 





50. From the amount shown above, you spend the amount shown below. 

Name the three coins you still have left. 

36. Write the names of the following coins: a thrup 'ny bit, tuppence, 

and a hay-p'ny. 

37. In symbols, write the total value of the coins shown. 
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38. In symbols, write the amount shown here. 

39. If one-and-six were taken from the coin shown here at the left, 
which single coin would you get in change. A, B, C, or D? 

40. The amount shown is taken from the coins shown at the left. In 
symbols, write the amount you should receive in change. 

41. To pay the amount shown, you give a florin and a shilling. Which 
coins. A, B, C, or D are your correct change? 

42. You give the coins shorn to pay the amount shown. In words, write 
the value of your change. 

43. You give the coins sho^ra to pay the amount sho^<m. In words, write 
the value of your change. 

44. You owe a friend the amount shown. As part pa3nnent, you give the 
coins shown. Which coins. A, B, C, or D do you still owe? 

45. You have six shilling. You spend half— a— crown. Which coins. A, B, 
C, or D do you still have left? 

46. You have the amount shown, and you spend a florin and a thmp*ny 
bit. In symbols, write the amount you still have left. 

47. You give a half-crown and a shilling to pay a bill for two-and- 
nine. In symbols, write the amount of change you get. 

48. From the amount shown here, you spend the coins shown here. In 
words, write the amount you have left. 

49. From the amount shown above, you spend the amount shown below. 
Which of the following coins. A, B, C, or D, do you have left? 

50. From the amount shown above, you spend the amount shown below. 

In symbols, write the amount you have left. 
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ENGLISH MONEY TEST 



QUESTION 


ANSWER 


36 


In words , write the 

total value of these / 




37 


In words, write the total value of the 
coins named here. 

1 HALF CROVJN + 1 FLORIN + 1 THREEPENNY BIT = 




38 . 

9 




33 


In words, write the value of a FLORIN. 






With these coins you pay 
two -and -nine. The clerk 
hands you one coin in 
change. What is the name...---.^^^^ 
of that coin? 

^ -V— 






To pay one-and-three you hand the clerk 
a HALF CROWN. In words, write the value 
of vGur chanae. 




^ You owe a friend six shillings, nine pence. 
YouL^ive him these coins as part payment. 

What single coin is still needed to pay him 
in full? 


s 
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From the total amount sho^ra here, you spend 
the coins shown here. In S3nnbols, writ^ 
the amount you have left. - 



14 SHILLINGS, NINE PENCE - 







You have four shillings, six pence in your 
pocket and you spend half-a-crown. What 
is the name of the single coin you have 
left? 







ENGLISH MONEY TEST II 



36 



QUESTION 

•i 



ANSWER 



37 



In S 3 nnbols, \7rite the total value of the 
coins shown. 

6 HALFCROt^NS + 3 FLORINS + 2 SHILLINGS + 6 PENCE = 



38 



In S 3 nnbols, \7rite the amount shown here, 
8 shillings, 2 pence, halfpenny 



39 



If one-and-six were taken from the coin shown 
here, which single coin would you get in 
change ? 




40 



The amount shown is taken from the coins 
shown at the left. In S 3 nnbols, write the 
amount you should receive in change. 




three -and- three 



To pay the amount shown you give a FLORIN and 
a SHILLING, Which coins are your correct change? 
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You give the coins shown to pay the amount 
shown. In words, x-nrite the value of your 
change , 



- 2 shillings, halfpenny = 




You give the coins shown to pay the amount 
shown. In words, write the value of your 
change . 



’r • 




- 2/11 



44 



You owe a friend the amount shown. As part 
payment you give the coins shown, I'Jhich 
coins do you still owe? 







0 
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45 



You have six shillings. You spend half-a-cro'tTn. 
Which coins do you still have left? 




You give a HALF CROWN and a SHILLING to pay 
a bill for two-and-nine . In sjTnbols, write 
the amount of change you get. 
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From the amount shown here, '• 

you spend the coins shown 
here. In words, write the - 
amount you have left. ' 
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APPENDIX B: 



INSTRUCTION BOOKLET 



You are going to plan a 15-minute lesson on the subject of 
English Money for sixth graders to supplement an existing slide-tape 
program. The existing program, plus your videotaped supplement, will be 
presented to some sixth-grade classes in the area. A number of teachers 
are working on this same project. Your job, through your videotaped 
lesson, is to teach the children what you think they need to learn to 
pass an achievement test on the subject immediately following the 
lesson. We are interested to learn which teaching strategy is most 
effective as measured by the test scores following the lesson. 

In this booklet you will find the following materials to 
help you in this task: 

1. A general statement of the learning tasks; 

2. A description of the stimulus-response elements 

as they are managed within the slide-tape program; 

3. A “test-item generator" which describes the kinds 
of test items which are likely to appear in an 
achievement test on the subject; 

4. A script of the slide-tape program for your reference 
and review; 

5. (Treatment II only) A copy of the final achievement 
test showing the proportion of sixth graders who 
answered each item correctly after a trial run of 
the slide- tape program. 

You will have a maximum of six hours to complete your tape 
from the time you start. You should allow no more than three hours for 
preliminary planning in order to provide sufficient time for rehearsing 
and recording portions of the lesson in the studio. Our staff will be 
in the area at all times to assist you in using the equipment, 
rehearsing the lesson, or to answer questions of a technical nature. 

Good Luck. 
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General Statement of the Learning Tasks 



The student encountering English Money for the first time is 
generally exposed to the actual coins and bills in circulation, to price 
tags and labels, and to merchants who quote prices and label pieces of 
money in colloquial speech. The tasks the student is called upon to 
perform in the presence of these aural and visual symbols involve 
computing equivalences and differences within the English money system 
and expressing the result in terms of several symbol systems 
interchangeably. A typical purchase might involve a merchant quoting 
a price in colloquial speech, the selection of an appropriate set of 
coins by the student to cover the purchase, the computation of the 
correct value of change to be received, and the assessment of equivalence 
between the value of the expected change and the value of the change 
received . 



The program proposes to expose the student to the expected 
range of stimulus classes asking him to make the typical responses he 
would be expected to make under "realistic" circumstances. Thus, in the 
presence of a variety of stimulus situations the student will be asked 
to express equivalences and differences in value in any of the 
conventional symbol systems. 

For administrative purposes, spoken responses are excluded 
from the program. 



General Objectives Established for the Program 

The following general objectives are established for the 

program: 

1. Recognition and labeling of English coins in common circulation. 

2. Computing equivalences and differences of value interchangeably 
among coins, written and spoken verbal statements, and numerical 
expressions according to the common practice. 

3. Expressing values interchangeably in terms of coins, written verbal 
statements, and numerical expressions according to the common 
practice. 



General Objectives NOT Established for the Program 

The following statements describe certain general objectives 
specifically excluded from those established for the program: 

1. Computing American monetary equivalents of British units. 

2. Estimating costs of common items in British commerce. 
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3. Spoken responses of any sort on the part of the student. 

4. S-R combinations involving coin-bill configurations valued in excess 



Stimulus Classes 

The student will be presented photographic representations of English 
coins and their respective labels as follows: half-penny, three-penny 

bit, shilling, florin, and half-crown. He will be presented a range of 
values from 1/2 d. to several pounds expressed in terms of written and 
spoken words, and numerical expressions; from 1/2 d. to 19/11 1/2 
expressed in terms of coins as well. 

Response Classes 

The student will be asked to respond to these stimuli by expressing 
equivalences and differences in value. These responses are to take the 
form of written verbal statements, numerical expressions, and selected 
photographs showing coin configurations. For ease of administration, 
spoken responses are excluded from the program. 



of 




The following table defines the stimulus-response elements as they are 
managed within the program. 



Program Element 


As Stimulus 


As Response 


Coin Configuration 


A photograph of coins 
is used as a stimulus 
pattern. 


S selects from several 
alternative photos of 
coins to express 
equivalence or 
difference. 


Coin Labels, 
spoken 


The names of coins 
are spoken colloqui- 
ally to S (eg. 
"Thrup*ny Bit"). 


speaks the names of 
coins colloquially. 
However, spoken re- 
sponses are excluded 
from this program. 


Coin Labels, 
written 


The names of coins 
are written out 
correctly spelled 
(ie. not as colloqui- 
ally pronounced* eg. 
"Three-Penny Bit") . 


^ writes the names of 
coins as they would be 
correctly spelled 
(ie. not as colloqui- 
ally pronounced, 
though spelling is 
NOT an objective of 
the program) . 


Value Statements, 
spoken 


A value is spoken to 
S colloquially (eg. 
"Tuppence Hayp*ny"). 


^ expresses a value 
colloquially . However , 
spoken responses are 
excluded from this 
program. 


Value Statements, 
written 


A value is written 
in words (eg. "Two 
Pence Half-Penny") . 


S expresses a value 
in words by writing 
them as a response 
(eg. "Two Shillings"). 


Value Expressions 


A value is expressed 
in terms of numerical 
symbols according to 
the conventional 
practice (eg. "12s. ld= 
12/1"). 


^ expresses a value 
in numerical symbols 
according to the 
conventional practice 
(eg. "6/5=6s.5d."). 
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BEHAVIORAL OBJECTIVES EXPRESSED 



AS TEST-GENERATING STATEMENTS 



^ is given any of the 
following: 


And is asked to 
subtract any of 
the following: 


And to express his 
answer by any of the 
following: 


- The names of some 
coins (written out) . 


- The value of some 
coins named in 
writing. 


- Naming a correct 
set of coins in 
writing. 


- The names of some 
coins (spoken 
aloud) . 


- A given amount 
of money spoken 
aloud . 


- Stating in words 
(written) a correct 
amount of money. 


- A set of coins 
(in photographic 
form) . 


- A given amount 
of money written 
out in words . 


- Expressing a correct 
amount of money by 
writing a conven- 
tional mamerical 
expression. 


- An amount of money 
(written as a 
conventional 
numerical expression) . 


- An amount of money 
written as a con- 
ventional numerical 
expression. 


- Selecting a correct 
set of coins from 
several sets shown 
in photographs. 


- An amount of money 
(written in words) . 


- The value of a 
given set of coins 
(in photographic 
form) . 


(Oral responses 
excluded) 


- An amount of money 
(spoken aloud) . 


- The value of some 
coins named aloud. 
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SCRIPT OF BASE PROGRAM 



ENGLISH MONEY 

A Fixed“Pace Audio-Visual Program 



This is a lesson about English money. 



to you. Also, I will ask you some 
questions as we go along. The question 
slides will look like this. 



Notice the figure in the upper- left 
corner. Whenever you see a question 
slide, write your answer in the correct 
space on your answer sheet. Try to 
answer every question. It will help 
you learn. Answer this question in the 
"A" space on your answer sheet. 

You should have put a "4" in answer 
space "A" on your answer sheet. If you 
have any questions, raise your hand now. 

(STOP TAPE — THEN START) 

Now let’s see hov7 much you can learn 
about English money . 

English money seems strange at first 
because it is not based upon the 
decimal system. 



The basic unit of English money is the 
Pound . 



VISUAL 



AURAL 




To teach you about English money I am 
going to show you some slides as I talk 
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Here is your first question. Write your 
answer in space number 1 on your answer 
sheet. v-Jhat is the basic unit of 
English money? 



The pound is the basic unit of English 
money. The symbol shown here is the 
symbol for pound. Draw this symbol 
three times in the scratch area of your 
answer sheet. 



This is how we write the symbol for one 
hundred dollars. Now for question 
number two. How would you write the 
symbol for one hundred pounds? 




This is how to vzrite the symbol for one 
hundred pounds. Notice that the pound 
sign comes before the amount. 



This coin is called a shilling. 





If you gave a one pound note for some- 
thing costing fifteen shillings, you 
would get five shillings in change. 

How many shillings must there be in 
one pound? 
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Th6re are twenty shillings to the pound. 
The syiabol for shillings is the small 
placed after the amount. 



”s" 




How would you write the symbols for one 
pound five shillings? 




? 



This is how one pound five shillings is 
written in symbols, notice that a 
period is placed after each denomination, 



How would you write in words the amount 
shown here? 

(PAUSE) 

You would write this as two pounds eight 
shillings. Sometimes a price is shown 
in shillings alone. 






Whenever a price is shown in shillings 
alone, you will see this symbol. Draw 
this symbol three times in the scratch 
area of your answer sheet. 



The symbol shown here is read nine 
shillings . 



How would you write the symbols for 
five shillings alone? 
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This symbol shows five shillings alone. 



How would you write the symbols for 
this amount in shillir.gs alone? 



Twenty-five shillings. Remember, there 
are twenty shillings to the pound. So 
twenty shillings plus five shillings 
gave us the twenty-five shillings shown 
here. 




tOSHILLINCSi 

10 /- 



If you gave two pounds for something 
costing one pound ten shillings , how 
much change would you receive? Write 
your answer first in words, then in 
symbols . 



You would receive ten shillings in 
change . 




This coin is a penny. 









9 ? 

I SHiLUNC 
-1 PENCE 



5 PENCf 



lld.= IS. 



(30 3. lOS. lid 



The plural of penny is pence. For 
example, you can buy a packet of 
gum for two pence. 



If you gave one shilling for something 
costing seven pence, you would receive 
five pence in change. How many pence 
must there be in a shilling? 



There are twelve pence in one shilling. 
Note that pence are expressed by the 
symbol ”d|V 
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There are twelve pence in one shilling. 



You could pay for a magazine priced at 
one shilling three pence with pence 
alone. How many pence would you need. 



You would need fifteen pence. Notice 
how the slash is used here to show both 
shillings and pence. 



How would you use the slash s3nnbol to 
vrite the amount shown here? 



3S. 6d. 



3/6 



This symbol means three shillings 
sixpence. In England it is usual to 
omit the words shilling and pence 
when speaking of sums of money. So 
this symbol is read as three and six. 
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15 ? 

1/4 . 





n ? 

46/11 
i,l. 3S. 9d. 





Write how you would read the 33^0001 
shown here and what it means? 



(PAUSE) 

The symbol is read as seven and four. 
It means seven shillings fourpence. 



If a shopkeeper told you that a pound 
of apples cost one and four, how 
much would you pay him for the apples? 



You would pay him one shilling 
fourpence. 



If one pair of slippers was marked 
forty— six and eleven and another was 
marked two pounds three shillings nine- 
pence, which pair would cost the least? 
(PAUSE) 

Two pounds three shillings ninepence is 
less than forty-six and eleven. 

As its name suggests, this coin is 
worth half a penny. The English pro- 
nounce the name of this coin the hay— 
p’ny. Altogether, how many hay-pence 
are there in a shilling? 

(PAUSE) 

There are twenty-four hay-pence in a 

shilling . 

This is the way to write the symbol 
for a hay-p*ny. 
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This gymlSnl would be read as six pence 
Iiay-p*ny. 



In words, write the amount shown here. 
(PAUSE) 

In words, this amount is read as one 
pound, five shillings, three pence, 
hay-p*ny . 



e/ioi 



lid. 




d-. 



X3 




In words, write the amount shown here. 
(PAUSE) 

This amount is read as six shillings, 
tenpence , hay— p ' ny . 



In England, if you bought something for 
tuppence, hay-p'ny, shown here, you 
could give the clerk three coins and 
not wait for change. What three coins 
make t.upp ence , hay-p ' ny ? 



Two pence plus one hay-p'ny make 
tuppence hay-p ' ny , 



One postcard costs thrupence hay-p'ny. 
For three postcards, how much would 

you have to pay? 

(PAUSE) 

You would pay tenpence hay-p'ny. 
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If you gave one shilling for something 
costing tenpence hay— p'ny, how much 
change should you get? 



You should get penny hay-p'ny in 
change . 



In England this coin is called a 
thrup'ny hit. It is worth 
thrup 'nee. 



In London you could buy a newspaper for 
thrup 'nee and pay the newsboy with one 
thrup'ny bit. But you could give him a 
shilling and get change. How much 
change should you get from a shilling? 
(PAUSE) 

You should get ninepence in change. 

If you give a shilling for a pint of 
milk costing ninepence, what single 
coin is your correct change? 

(PAUSE) 

A thrup'ny bit would be correct change. 



These three coins of lowest denomina- 
tion are the hay-p'ny, the penny, and 
the thrup'ny bit. In England they are 
called coppers . 
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This small coin is a sixpenny bit. It 
is the lowest denomination silver coin, 
If it is worth sixpence, how many six- 
penny bits are in a shilling? 

(PAUSE) 

There are two sixpenny bits in a 
shilling . 




This silver florin is often called a 
two shilling piece. How many florins 
are in a pound? 

(PAUSE) 

Ten florins make a pound. Remember, 
there are twenty shillings to a pound. 



I FLORINS ?d. 







HALF CROWN: 

. 1/6 



30 



How many pence are there in a florin? 
(PAUSE) 

There are twenty-four pence in a florin. 



These coins do not quite add up to a 
florin. What single coin could you add 
to make up the correct amount? 

(PAUSE) 

An additional sixpenny bit is needed to 
make two shillings. 



This coin is a half crown. It is the 
largest silver coin in circulation. A 
half crown is worth two shillings 



sixpence. 



or two-and-six. 



These coins do not quite add up to a 
half crown. What single coin could 
you add to make up the correct amount? 

(PAUSE) 

An additional shilling is needed to 
make two-and-six. 
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These coins do not quite add up to a 
half cro^ra. How many thrup'ny bits are 
needed to make up the correct amount? 

(PAUSE) 

Two thrup'ny bits are needed to make 
two shillings sixpence. 



You ask for change of a half crown in 
sixpenny bits . How many sixpenny bits 
will you get? 

(PAUSE) 

You will get five sixpenny bits for 
your half crown. 



You give these two coins for a ticket 
costing two-and-six. How much change 
should you get? 

(PAUSE) 

You should get sixpence in change. 



These are the four silver coins of 
England. Added together, how much are 
they worth? 

(PAUSE) 

Altogether they are worth six shillings 



How much is this pocketful of change" 
worth? 

(PAUSE) 

Altogether this change is worth six 
shillings fourpence ha'p'ny. These are 
all the coins now in circulation. This 
is the end of the picture lesson. 






ENGLISH MONEY TEST 



■ 

From the amount shown above j you spend the amount shown 
below. Which of the following sets of coins do you 
have left? A, B, C, or D? 



5 shillings, 4 pence 
- 3 shillings^ 9 pence 




A 



B 



C 



D 




If one-and-six were taken from the coin shown here at 
the left, which single coin would you get in change? 




To pay the amount shown here at the left, you give a 
FLORIN and a SHILLING. Which set of coins is your 
correct change? A, B, C, or D? 




You have 4 shillings, 6 pence in your pocket and you 
spend half-a-crown. What is the name of the single 
coin you have left? — — 
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In words, write the value of a FLORIN. 



You owe a friend the amount shown. As part 

pa 5 nnent you give the coins shown. Which 

coins do you still owe? A, B, C, or D? / 




Ll 



_A BCD 

From the amount shown here, you spend the coins 
shown here. In words, write the amount you 
have left. 



0 




From the total amount 
coins shown here. In 
you have left. 



14 SHILLINGS, 9 PENCE 



shown here, you spend the 
S 5 mibols, write the amount 
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With these coins you pay two-and-nine . The clerk 
hands you one coin in change. What is the name 
of that coin? 




E 






END OF TEST 



I am a • 

(boy or girl) 

My birthdate is — ■■ — r \ 

(Month) (day) (year you were born) 

I thought this lesson was fun s ort-of-OK hori^ 

(circle one) 

This is how you could make this lesson better. 
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ENGLISH MONEY PROGRAM 
TRYOUT DATA 



Test Item // 


Correct Response 


% of Sample 
I'Jho Answer 
Correctly 


1 


C 


.50 


2 


B 


.43 


3 


C 


.37 


4 


Florin, 2s . 


.27 


5 


2 Shillings , 2s . , 
24d. 


.20 


6 


A 


.20 


7 


3s. 5d., 3/5 


.17 


8 


12s. 3d., 12/3 


.17 


9 


3-Penny Bit, 3d. 


.17 


10 


^2. 6d. 4%d. 


.13 



Average percent correct in sample group - 22.00+ 
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ENGLISH MONEY PRACTICE WORKSHEET 

i 

i 


Try to answer every question. Keep up with the filmstrip. Do NOT 
try to correct your mistakes. This is not a test. 


SCRATCH AREA 


A 


18 




1 


19 




2 


20 




3 


21 




4 


22 




5 


23 




6 


24 




7 


25 


V. 


8 


26 


9 


27 


10 


28 


11 


29 


12 


30 


13 


31 


14 


32 


15 


33 


16 


34 


L7 


35 



DO NOT TURN PAGE UNTIL DIRECTED 
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